Universal Access To All Knowledge
Home Donate | Store | Blog | FAQ | Jobs | Volunteer Positions | Contact | Bios | Forums | Projects | Terms, Privacy, & Copyright
Search: Advanced Search
Anonymous User (login or join us)
Upload

Reply to this post | See parent post | Go Back
View Post [edit]

Poster: Administrator, Curator, or StaffJonathan Aizen Date: May 4, 2003 5:04am
Forum: etree Subject: Re: Putting email addresses in the details pages

The /audio part of this site isn't crawled because it's excluded in the robots.txt - I think this is a remnant and should be changed. If changed, the section will be heavily crawled by Google - they currently crawl the forums and other parts of the site multiple times per day.

Info files (txt files on the etree0x servers) are actually crawled already because those machines don't have robots exclusions.

Reply to this post
Reply [edit]

Poster: glenn Date: May 7, 2003 1:29pm
Forum: etree Subject: Re: Putting email addresses in the details pages

there do exist user-agents which ignore the robots.txt exclusion file, so it is safe to assume that all portions of this site can be crawled by malicious (email-address-harvesting) user-agents.
A better place to put seeder's email addresses, in my opinion, is in the info.txt file included with the seed.

Reply to this post
Reply [edit]

Poster: Administrator, Curator, or StaffJonathan Aizen Date: May 7, 2003 2:24pm
Forum: etree Subject: Re: Putting email addresses in the details pages

Again, the text files are crawled, so that won't help...